The invention is useful for transmitting digital data such as management and control data out of band on a subchannel carrier transmitted on the same media as high speed data on a network or digital telephone line without interference therewith. The subchannel carrier uses a portion of the bandwidth that is not heavily populated by high energy frequency components of the high speed data traffic.
In many large computer systems such as are found in banks, at credit card transaction processing centers; etc., huge amounts of data must be moved and stored. Typically, very large disk arrays are used to store the data and these disk arrays are connected to file servers. These type systems require of the server/disk connection high reliability, high speed, large throughput and large bandwidth since huge amounts of information are being processed and a system shutdown or slowdown adversely impacts customers (and possibly safety in the case of large air traffic control computer systems). Because these systems can be spread out over several buildings or even if all the servers and disk arrays are within the same building, the easiest way to interconnect all the servers and disk drives for maximum redundancy is through a local area network usually with a hub having drop lines connected to all servers, disk drives and other units. Because disks fail frequently and must be replaced by redundant disks, there is a large amount of management and control traffic that must go back and forth over the server/disk array connections and through the hub to the management and control process. This management and control traffic is necessary to determine which disks are on-line, where those disks are on the network, operational disk status, how many fans are still running (because when a fan fails, the disk it cools will probably be next), what servers are still operational, network fault status, network traffic conditions and statistics, etc.
Most if not all such digital data transmission systems require the bidirectional transmission of digital management and control data between nodes to collect data regarding the performance of the system and manage the various nodes, bridges etc. in the system. The out of band management and control data will be hereafter referred to as subchannel data.
There are many ways of modulating subchannel data onto a high speed digital signal, many of which are taught in a prior co-pending application of the assignee of the present application, which is hereby incorporated by reference. However, many of these prior art methodologies will only work in some systems and not in others. For example, modulating the subchannel data onto the clock by phase or frequency modulation will cause too much jitter in some systems, and also requires access to the high speed transmitter and receiver clock generation and clock recovery circuts. This is not possible in all cases, so a methodology that does not requires such access is preferred. The embodiments disclosed herein do not require such access. Amplitude modulation of the data with the subchannel modulation will not work in systems where digital buffers stand between the subchannel transmitter and the transmission media such as occurs where the high speed data media is fiber and a digital electronic-to-light media driver transducer is used to convert the digital high speed data to light signals.
Out of band management topologies get management data to and from the hub easily enough on separate network segments, which has its own set of problems described below. However, getting management and control data to and from the disk arrays in-band is more difficult. The management and control data sits in registers on a board in the disk array. In order to get this low speed, low priority data onto the high speed data path, special circuitry must be built in each disk array which interfaces these registers to the high speed data path. This circuitry functions to collect and format the management and control data into the type of data packets used in the high speed data path and to transmit these packets with the correct communication protocol to the hub. Since the management and control data does not consume an entire packet of the size used to send data on the high speed data path, some packet Space is empty and wasted. The need for this special circuitry to put the management and control data in the high speed data path makes the disk arrays more expensive and complex and placing management and control data in the high speed data packets wastes network throughput.
Some disk arrays store the management and control data on a separate disk drive which can be polled. However, this approach does not solve the problem of the need for special circuitry to get the management and control data into packets in the high speed data path and the resulting extra complexity and wasted throughput. It only allows time shifting thereby enabling transmission of the management and control data when network traffic volume is low and the throughput loss is not as significant. This time shifting is implemented by providing more storage capacity for the management and control data than is provided in the registers of the other prior art type disk arrays described above.
In some networks, management traffic is transmitted in-band by placing the management and control packets inside empty data packets and shipping these partially filled packets over the existing network connections between the server, hub and disk drives. This causes loss in throughput since the data packets are large and the management data does not fill the data packets entirely. This leads to wasted bandwidth. Further, arbitration by the management process to have one or more packets awarded to it for management traffic and to have access to the network so that transmissions to all servers and disk drive arrays of management messages consumes processing resources and network throughput unnecessarily.
One proposal has been made in the prior art to interleave special management and control packets in with the packet stream on the high speed data path to alleviate the above mentioned problem. However, this makes the design of the integrated circuits that implement the processing on the various network protocol layers more complex and difficult. So far this approach has been a commercial failure.
Another approach that has been tried in the prior art for transmission of management and control packets is to provide an entirely separate network for the management and control data such that each server and disk array is connected to the out of band management and control process running on a separate diagnostic processor by its own network segment. This substantially increases the wiring and connection cost of the system, especially in distributed systems, as every server and disk array must have an additional network segment connected thereto. In addition, each server and disk array uses a card slot for the management and control network card which adds to the expense, complexity and failure point count of each of these units.
A multiplexing approach that has been used in the prior art to send multiple television signals over the same media is represented by U.S. Pat. No. 3,623,105. This patent teaches receiving multiple video signals and translating each one to a different channel or subband and adding all the subband signals together to form a composite signal. The composite signal is then applied to the frequency control input of a VCO having a nominal frequency of 750 mHz. The output of the VCO is applied to one input of a pulse width modulator the other input of which receives an 18 gHz carrier. The pulse width modulator serves to key the 18 gHz carrier in accordance with the period of the signal from the VCO so that what is transmitted is a train of 18 gHz waves the width of each wave or xe2x80x9cpulsexe2x80x9d being set by the instantaneous period of the signal output from the VCO.
Another approach to implement a subchannel or auxiliary channel over a digital communication system is represented by U.S. Pat. No. 4,079,203 to Dragoo. This patent teaches an auxiliary channel implemented on a time division multiplexed carrier system by modulating the pulse repetition rate of the digital bitstream on the transmit side of the transaction. Each transceiver transmit section includes a FIFO shift register acting as a buffer. Modulation of the auxiliary channel data is carried out by varying the rate at which the digital information of the main channel is clocked out of the FIFO shift register at the transmitting end. The modulating signal varies the pulse repetition rate of a voltage controlled multivibrator which has its output coupled to the clock out input of the FIFO shift register and to a phase comparator of a phase lock loop. Data is clocked in using a clock signal derived from the incoming serial data stream. One drawback of this system is that the capacity of the FIFO can be exceeded if the clock out rate falls substantially behind the clock in rate because of the nature of the modulating signal during certain intervals.
Another approach which has been tried in the prior art is represented by U.S. Pat. No. 4,425,642 to Moses et al. This patent teaches sending digital data simultaneously with analog signals over the same media used by an analog signal communication system such as a telephone or television video. This is done by converting the digital data into very low power multifrequency signals consisting of fundamentals and harmonics. Although the harmonics are in the frequency range of the main signals, their low power does not cause substantial interference. Complicated filter arrangements at the receiver end separate out the harmonics and fundamentals that encode the digital signals and demodulation circuitry at the receiver decodes these frequencies back into digital data. This approach is not well suited to a system where the main data flow is digital and requires complex filtering and decoding arrangements at the receiver and is therefore unduly expensive and complex.
Another approach that has been tried in the prior art is exemplified by U.S. Pat. No. 4,677,608 to Forsberg. This patent teaches a method of implementing a service channel over a fiber optic system line which carries high data rate, e.g., 34 megabits/sec, data encoded with a code the power spectrum of which is heavily suppressed for low frequencies. The low frequency section of the media bandwidth is used to transmit a service channel with a low bandwidth. The service channel signals are frequency modulated, and this frequency modulated signal carrying the service channel data is then used to pulse width modulate pulses forming the high speed data stream.
It is critical to note in fully understanding the invention that this Forsberg scheme only transmits high speed data in the RZ or return to zero format on the fiber optic media, and, if NRZ format high speed data is input to the pulse width modulator, the NRZ format high speed data is converted to RZ format high speed data for transmission on the fiber optic media by performing an AND operation with the clock, as shown in FIG. 5 (see Col. 3, lines 58-61). Either way, the RZ data is a stream of pulses (during logic 1 bit times only) that are pulse width modulated by the subchannel carrier signal. RZ format high speed data is not compatible with standard high speed data receivers designed in accordance with the Fibre Channel and Gigabit Ethernet standards which are the network environments in which the various species of the invention disclosed herein are intended to work. Also, with RZ data, the number of transitions from one logic state to another is approximately double the number of transitions of NRZ data over the same number of bit times. Forsberg""s system pulse width modulates each pulse representing a logic 1 so he gets perturbations of the time of crossing a reference voltage for both a rising edge and a falling edge of that pulse. With RZ data, if two consecutive logic 1""s are followed by two consecutive logic 0""s over four bit times, there will be a pulse with two transitions in each of the first two bit times (representing two consecutive logic 1""s) and no pulses in the next two bit times. In contrast, with NRZ data for the same hypothetical, there will be a rising edge at the beginning of the first bit time and a falling edge at the end of the second bit time and no transitions during the third and fourth bit times.
The critical distinction is that NRZ data has a transition density which is variable over time depending upon the content of the data even if the data is D.C. balanced, while D.C. balanced RZ data does not have a variable transition density. RZ data is also incompatible with Fibre Channel networks. Since the subchannel information in Forsberg""s system is expressed in the form of perturbations of the time of crossing the reference voltage on each transition of all logic 1 pulses, his system has a known and reliable number of transitions in which to send subchannel energy. Therefore, Forsberg does not have a variable transition density problem, and teaches no circuitry to deal with the fact that where pulse width modulation is in used to carry the subchannel information, transition density variations vary the magnitude of the subchannel signal at the receiver. Because RZ data is used exclusively in the Forsberg system as the data format on the media, a much stronger and nonvarying amplitude subchannel signal is available at the subchannel receiver which is easier to detect in the presence of noise.
Forsberg""s system apparently was designed to work in telephony systems built by Ericsson (the assignee of the Forsberg technology) which are not transmitting One Gigabit digital data on a local area network. This conclusion can be drawn from the teachings at Col. 2. line 65 where the subchannel data rate is indicated to be 0.3-4 kHz, and Col. 3, line 6 where the FM modulated subchannel data signal is indicated to be 35-65 kHz, and from Col. 2, line 54 where the high speed bit rate is taught to be f0 and Col. 3, lines 21 and 22 where the clock rate is taught to be f0=1/T and Col. 3, lines 35-36 where the clock frequency is taught to be around 40 MHz.
The prior art subchannel transmitter described in co-pending application Ser. No. 09/063,633, filed Apr. 20, 1998, entitled SUBCHANNEL MODULATION SCHEME FOR CARRYING MANAGEMENT AND CONTROL DATA OUTSIDE THE REGULAR DATA CHANNEL solved these problems by using a form of pulse width modulation implemented by summing the high speed data with frequency shift keyed subchannel data modulated onto a carrier having a frequency that is much lower than the 1 GB high speed data. The frequency shift keyed carrier signal instantaneous amplitudes were added to the 1 GB data stream, but since the carrier frequency of the frequency shift keyed subchannel carrier was so much lower than the 1 GB data rate, the effect was as if a reference voltage level from which the logic 1 and logic 0 levels were measured was being changed at a slow rate relative to the high speed data rate. This raised and lowered the logic 1 and logic 0 levels of the high speed data in accordance with the lower frequency subchannel data, but these raised and lowered logic 1 and logic 0 levels were lost when the high speed data signal was passed through a digital buffer. However, the raised and lowered levels also changed the timing of zero crossings of the high speed data, so the subchannel data survived the digital buffer as pulse width modulation in the form of the perturbed timing of the zero crossings of the high speed data. Low pass filtering and pulse width demodulation techniques are used in the receiver to recover the subchannel data since the carrier frequency of the subchannel lies in the range of frequencies where the spectrum of the high speed data does not contain much energy since 8B/10B encoding or other DC balanced encoding schemes are used for the high speed data.
While this system works well, receiver sensitivity can be improved, and the subchannel transmitter and receiver can be made both smaller and cheaper by using digital techniques. The original subchannel receiver was relatively inexpensive since it was constructed with standard off the shelf ceramic filters that had a passband of about 110 kHz. Since these were readily available, they were inexpensive. However, that bandwidth is vastly greater than is needed to receive the relatively slow baud rate of the subchannel data. Thus, a need arose for a receiver design with a more narrow bandwidth that could be made more selective so as to reject more of the spurious noise in the frequency range of the subchannel signal. Narrower receiver passband characteristics allows subchannel signals with worse signal-to-noise ratios to be received.
Also, the original subchannel transmitter used an expensive numerically controlled oscillator and a high-speed digital-to-analog converter and was not a one chip design so it was relatively expensive to build. The reason this particular analog design was selected was for flexibility purposes because it allowed the subchannel carrier frequency to be programmed. At that time, it was unclear what the subchannel carrier frequency should be to get best performance in the Fourier spectrum of the 8b/10b encoded high speed data. Certain GBICs and other third party vendor equipment that the subchannel signal had to pass through would attenuate the subchannel carrier more than other equipment. This was a problem because the subchannel carrier signals does not have much more power than the frequency components in the same frequency range caused by the 8b/10b encoded high speed data signal. The signal power of the subchannel carrier signal cannot be increased at will to improve the signal-to-noise ratio, because to do so causes greater pulse width modulation excursions which are interpreted in the high speed data receivers as jitter. Excessive jitter in the high speed data receivers can cause an intolerable bit error rate, and, in particularly bad cases, can cause loss of clock synchronization.
However, experience in actual operations in the field has brought to light the best frequency for the subchannel carrier. Further, the frequency deviation between the Mark and Space carrier frequencies that propagate best through the third party vendor components in the high speed signal path is better understood. Thus, there is a need for a subchannel transceiver design which can also be made both less expensive and smaller. Preferably, this subchannel transceiver is implemented as a one chip digital design with no programmability of the subchannel carrier frequency but a single chip is not absolutely required. Also, lack of programmability of the subchannel carrier frequency is not essential to the invention, and transceivers with programmable carrier frequencies are intended to be within the scope of the claims appended hereto.
Further, with the prior subchannel transceivers, much attention had to be paid to adjusting the transmit levels of the analog frequency shift keyed subchannel signal to avoid causing excessive jitter and to compensate for manufacturing variations in the other components used in the high speed signal path. This need for scrutiny and adjustment for every installation is inconvenient and is labor intensive. Thus, a need has arisen for a subchannel transceiver structure with less of a requirement for adjustment at the site of the manufacturer of equipment into which the subchannel transceivers are included. However, since there are many analog systems still deployed in the field, any new subchannel transmitter and receiver structures for the subchannel data must be compatible with these legacy systems.
The genus of the invention defined herein is defined by the following structural characteristics.
First, there is a media such as fiber optic cable or copper coaxial cable or other copper media such as twisted pair which functions to carry high speed digital data from a high speed digital data transmitter to a high speed digital data receiver. High speed digital data in this regard just means a data stream on an Ethernet, Fiber Channel, ring or other network, telephony system or other system for delivering data other than subchannel data.
Second, there is a digital subchannel transmitter for sendin subchannel data which is a separate data stream from the high speed data and has a much lower data rate. The subchannel transmitter includes a modulator to modulate the subchannel data onto a host signal which can propagate across said media without excessive attenuation. The host signal can be a separate subchannel carrier, the clock embedded into the high speed data or the high speed data waveform itself. In the preferred species, the modulator frequency shift keys the subchannel data onto a subchannel carrier which has a frequency which is set in a portion of the spectrum of the high speed data where the amount of interference from the high speed data frequency components is not so high as to force the subchannel modulation amplitude to be so high as to cause jitter in the high speed data receiver which is excessive. In these preferred embodiments, the modulator includes a local oscillator in analog or digital form which generates a carrier signal at the subchannel carrier frequency. In one embodiment, the subchannel carrier frequency is just over 1 MHz, but in other more preferable embodiments that do not have to be concerned with compatibility with previous designs, the subchannel carrier frequency is between 0.5 and 0.75 MHz (assuming a one gigabit high speed data link). In alternative species, such as are disclosed in the parent application, the subchannel data is phase or frequency modulated by the modulator onto the high speed data clock. In the case of a fiber media, the modulator modulates the subchannel data onto the light intensity of the pulses generated by the laser diode by converting the subchannel data to a current signal and adding the current signal to the current signal generated by the high speed data or to the bias signal that is added to the current signal generated by the high speed data. The combined current signal is then used to drive the laser diode. Regardless of form of the host signal that is used to propagate the subchannel data across the media, the modulator uses a form of modulation and uses a sufficient amplitude given the characteristics of the host signal and the media such that the subchannel data can be detected in the presence of noise. The characteristics of the high speed data signal that are important to consider are the amplitude, rise time, transition density, encoding type (DC balanced?) or other characteristics of the high speed data. For example, transition density and rise time are important considerations in determining the proper amplitude of subchannel carrier signal to inject for a given media type in pulse width modulated embodiments so as to develop sufficient subchannel modulation intensity at the subchannel receiver such that the signal can be detected in the presence of noise without causing excessive jitter. In some species, the subchannel transmitter includes a variable attenuator which is used to control the amplitude of the injected subchannel carrier such that if a copper media is substituted for a fiber media, the amplitude of the injected subchannel carrier can be reduced so as to not cause excessive jitter and loss of sync of the high speed data receiver clock recovery PLL. This is made necessary by the differences in losses between copper and fiber media for the high frequency components of the high speed data signal and the lower frequency components of the subchannel data. In transmitters where the media type is known, and never will be changed, the variable attenuator is not necessary, and fixed attenuation elements or other circuitry can be used to generate the correct amplitude for the subchannel carrier signal. The modulator must use a modulation type and intensity such that the modulation carrying the subchannel data information will pass through said media and any associated drivers or transducers at the transmitter and/or receiver end of the connection without excessive attenuation such that the subchannel can be detected and recovered at the location of the subchannel receiver. The modulation type and intensity can vary depending upon the media and whether the combined high speed data and subchannel modulation need to pass through digital type drivers/transducers. Pulse width modulation works best if the combined signal must pass through digital drivers, but other forms of modulation such as clock phase or frequency modulation or laser diode intensity modulation can also be used if digital drivers are present. Where no digital drivers are present between the subchannel injection and extraction points, such as in copper media, the types of modulation that can be used by the subchannel transmitter include all known forms of modulation such as BPSK, QPSK, AM, FM, pulse position and pulse amplitude modulation, etc. The various types of modulation schemes and modulators are disclosed in: Lee and Messerschmitt, Digital Communications, 2d Ed., Kluwer Academic Publishers, Boston (1994) ISBN 0-7923-9391-0, TK5103.7.L44; and, Haykin, Communication Systems, 3d Ed., Wiley and Sons, Inc., New York, N.Y. ((1994) ISBN 0-471-57178-8, TK5101.H37, both of which are hereby incorporated by reference. In some embodiments, the subchannel transmitter includes a UART to serialize parallel format subchannel input data. In other embodiments, the UART has a selectable output baud rate. In other embodiments, the subchannel data is input in parallel format and is serialized by one or more parallel load, serial output shift registers in the subchannel transmitter.
Third, a subchannel receiver is needed to detect the subchannel data. The subchannel receiver in the genus defined herein can be either analog or digital, but it must have a much more narrow filter bandwidth than is currently available in commercial frequency shift keyed receiver chips or separate ceramic filters. The bandwidth of the receiver intermediate frequency section must be just wide enough to encompass the spectrum of the subchannel data so as to pass most if not all of these subchannel frequency components at whatever data rate is chosen for the subchannel data while rejecting substantially all of the high speed data frequency components outside the spectrum of the subchannel data. While the high speed data frequency components that overlap the subchannel frequency components in frequency will pass through this bandpass filter (usually located in the IF section of the receiver), there will be far fewer frequency components from the high speed data at frequencies above and below the subchannel spectrum which will be passed than in previous designs by the assignee. This makes the receiver more selective and sensitive thereby eliminating sensitivity to manufacturing variations in rise time of the high speed data signals in FSK embodiments. In embodiments where the subchannel data rate is selectable, the bandwidth of the receiver IF bandpass filter is also selectable so as to be wider for higher baud rate, subchannel data. In commerically available FSK integrated receivers with external bandpass filters for the IF section, the most narrow commercially available bandpass filter which is reasonable in price is an external ceramic filter which has a bandwidth of about 100 KHz (at standard FM frequencies). Very narrow external ceramic filters with bandwidths on the order from 4-12 KHz are available for AM receivers, but they are not useable in the subchannel application because the Mark and Space frequencies are 56 KHz apart, so these AM type filters are not wide enough to pass both the Mark and the Space frequencies. The 56 KHz spacing is needed in the preferred embodiment for compatibility with previous designs. However, if there is no prior system compatibility issue, a smaller deviation frequency can be selected, and the more narrow AM type analog receiver chips with the narrow bandwidth AM type ceramic filters could be used to achieve similar performance to the narrow bandwidth digital filters and digital demodulator design disclosed herein. Such an AM type analog receiver and demodulator would have the same structure as the analog receiver disclosed in the parent application, which is hereby incorporated by reference, but FSK modulation with smaller deviation frequency would still be used. At the deviation frequency chosen for the Mark and Space frequencies (Mark is 1.1808 MHz or 41*28.8 KHz and Space is 1.1232 MHz or 39*28.8 KHz) originally chosen for the analog receiver, the bandwidth of a filter that encompasses the spectrum of both the Mark and Space frequencies is only 57 KHz wide. Thus, the older design 100 KHz bandwidth let more noise from the high speed data frequency components in the vicinity of 1 MHz in than was necessary. The basic idea behind the improved receivers disclosed herein is to use a much more narrow bandwidth filter in the IF section of the receiver so as to reject more high speed data noise and make the receiver more sensitive and selective to the subchannel data signal only. This has two benefits. First, the more narrow filter bandwidth can be easily achieved with digital FIR filters at reasonable expense so an entirely digital receiver can be made which is completely integrated on an ASIC and this makes the receiver less expensive to build and have higher sensitivity and selectivity than the prior analog receiver design. The higher sensitivity and selectivity provides the second benefit which is elimination of the need for adjustments of the transmitter subchannel carrier magnitude of the injected signal and receiver tuning for every shipped hub or switch with a subchannel system in it to take into account the lot-to-lot manufacturing variability of the rise times of the high speed data signals. The greater sensitivity of the receivers with smaller IF filter bandwidth provides a larger window such that regardless of variations in the rise times of the high speed data from one unit to another, the minimum subchannel signal can be detected with no problem and the maximum subchannel signal will not cause problems with high speed data reception. Thus, variable attenuators are needed in the subchannel transmitters only where media type switches might occur. The same result can be achieved with an analog receiver of the design disclosed in FIG. 18 of the parent case, but a custom designed filter 292 with a bandwidth of only about 57 KHz must be provided (or whatever bandwidth is adequate to encompass both the Mark and Space frequencies for whatever deviation frequency is selected). If another form of modulation is selected such as QPSK etc., then the bandwidth of the filter in the receiver IF section must be just wide enough to pass enough of the frequency components to provide effective subchannel detection without letting unnecessary noise from the high speed data spectrum. This means most of the high speed data frequency components above and below the frequency of the subchannel spectrum must be rejected. Detection of the subchannel bits depends upon the type of modulation used. The detector will function to recover the subchannel bits by generating from the signal output by the bandpass filter one or more signals having characteristics which depend upon the subchannel data bits encoded in the host signal. In the case of FSK modulation, the detector operates by generating a single signal which has a voltage that depends upon whether a Mark or Space is being received at any particular time or it can generate two signals, one of which has a magnitude that is higher than the other when a Mark is being received and vice-versa when a Space is being received. In the case where the subchannel signal is used to frequency or phase modulate the clock embedded in the high speed data signal, the detector can take the form of a circuit that recovers the subchannel bits from the error signal generated by the phase detector in the phase lock loop of the clock recovery circuit in the high speed data receiver. This error signal will have a first characteristic when the subchannel bit is logic one thereby altering the frequency or phase of the high speed data clock in a first direction, and will have a second characteristic when the subchannel data bit is a logic zero thereby altering the frequency or phase of the high speed data clock in a second direction. If the subchannel data is modulated onto the high speed data signal itself by modifying the intensity of the light pulses generated for the high speed data in a first direction for a logic one and in a second direction for a logic zero, then the detector is an envelope detector that detects the changes in the amplitude of the envelope of the high speed data light pulses and converts these changes in amplitude to subchannel bits.